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1.  Introduction 


1.1  Background 

Because  Infantry  Soldiers  operating  ground  robots  typically  have  multiple  task  demands, 
reducing  the  tasks  involved  in  robotic  control  should  help  mitigate  potential  cognitive  overload. 
Maes’  (1995)  continuum  of  robot  control  that  ranges  from  teleoperation  to  full  autonomy  allows 
robots  to  be  placed  on  a  continuum  based  on  the  level  of  human-robot  interaction  that  is 
required.  Teleoperation  is  the  lowest  level  of  automation  on  the  continuum  because  it  requires 
the  most  intervention  from  the  robot  operator.  A  teleoperated  robot  is  totally  under  the  control  of 
the  operator  who  uses  a  joystick  or  other  control  device  to  command  the  robot.  This  requires 
constant  interaction  between  the  robot  and  the  operator.  Teleoperation  is  not  the  ideal  solution 
for  all  situations.  Many  teleoperation  tasks  are  repetitive  and  boring  and  the  work  requires 
constant  attention  by  the  operator. 

Semi-autonomous  control  (often  called  supervisory  control)  requires  the  operator  to  provide  an 
instruction  or  portion  of  a  task  that  can  safely  be  perfonned  by  the  robot  on  its  own.  Two  types 
of  semi-autonomous  control  are  often  identified — shared  control  (also  called  continuous 
assistance)  and  control  trading.  Shared  control  requires  the  teleoperator  to  delegate  a  task  for  the 
robot  or  to  accomplish  it  via  direct  control  of  the  robot.  If  the  operator  delegates  control  to  the 
robot,  then  he  or  she  must  still  monitor  the  robot  to  ensure  that  it  is  performing  the  task  correctly. 
A  guarded  teleoperated  robot  can  be  placed  on  this  portion  of  the  continuum  because  it  has  the 
ability  to  sense  and  avoid  obstacles  but  will  otherwise  navigate  as  driven,  like  a  robot  under 
manual  teleoperation.  During  control  trading,  the  human  only  interacts  with  the  robot  to  give  it  a 
new  command  or  to  dynamically  re-task  its  mission.  A  line-following  robot  can  be  placed  on 
this  portion  of  the  continuum  if  it  simply  follows  something  painted,  embedded  or  placed  on  the 
floor  and  does  not  have  the  ability  to  circumnavigate  obstacles  on  its  own. 

After  receiving  instructions,  autonomous  robots  (those  that  perfonn  tasks  independently  and/or 
have  the  capacity  to  choose  goals  for  themselves)  operate  under  all  reasonable  conditions 
without  recourse  to  an  outside  operator  and  can  handle  unpredictable  events  (Haselager,  2005). 
Huang  et.  al  (2005)  state  that  a  fully  autonomous  robot  has  the  ability  to  gain  infonnation  about 
the  environment,  work  for  an  extended  period  without  human  intervention,  move  itself  through 
its  environment  without  human  assistance,  and  avoid  situations  that  are  harmful  to  people, 
property,  or  itself  unless  those  are  part  of  its  design  specifications.  An  autonomously  guided 
robot  can  be  placed  on  this  portion  of  the  continuum  because  it  knows  at  least  some  infonnation 
about  where  it  is  and  how  to  reach  waypoints  (Murphy,  2000).  Knowledge  of  its  current  location 
is  detennined  by  using  sensors  such  as  lasers  and  global  positioning  systems  (GPS).  Positioning 
systems  detennine  the  location  and  orientation  of  the  platform,  so  the  robots  can  plan  a  path  to 
their  next  waypoints  or  goals.  Autonomous  robots  make  their  own  choices  instead  of  following 
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goals  set  by  other  agents.  A  robot  that  is  capable  of  goal  generation  can  also  be  placed  on  this 
portion  of  the  continuum. 

The  incorporation  of  autonomous  robotic  systems  into  military  schema  involves  more  than  just 
effective  system  design.  It  is  also  important  for  designers  to  understand  which  robotic  system 
autonomous  capabilities  match  or  surpass  human  abilities  and  when,  during  scenario 
accomplishment,  the  human  needs  assistance  or  is  overloaded.  This  knowledge  is  needed  in 
order  to  leverage  autonomous  robotic  behaviors  for  optimal  performance.  Thus,  the  goal  is  to 
optimize  the  human  and  robot  roles  in  a  task.  To  do  this,  robot  control  competencies  and 
inefficiencies  must  first  be  identified  and  then  they  must  be  understood  in  relation  to  specific 
task  performance.  Trade-offs  among  levels  of  autonomy  and  moderators  of  robot  control 
perfonnance  must  be  identified.  For  example,  workload  is  an  important  moderator  of  human 
task  perfonnance.  The  effectiveness  of  a  control  system  is  likely  to  depend  a  great  deal  on 
situational  task  demands.  The  requirement  for  the  operator  to  perfonn  simultaneous  tasks  such  as 
controlling  multiple  robots  or  performing  local  security  tasks  could  have  a  detrimental  effect  on 
human  intervention  in  robotic  task  outcomes  because  of  human  availability  at  a  certain  point  in 
time  and  because  of  cognitive  overload. 

Many  studies  have  demonstrated  that  operators’  situation  awareness  (SA)  was  higher  when  they 
were  controlling  robots  with  semi-autonomous  or  autonomous  capabilities  (Chen  et.  al,  2008; 
Dixon  et.  al,  2003;  Luck  et  al.,  2006).  However,  increased  autonomy  is  not  always  the  most 
effective  choice.  Chen  and  Joyner  (2006)  found  that  participants  detected  fewer  targets  when 
their  robot  was  operating  in  the  semi-autonomous  mode  rather  than  the  teleoperation  mode.  In  a 
two-year  study  of  a  collaborative  human-robot  system,  Stubbs  et.  al  (2007)  found  that  as 
autonomy  increased,  users’  inability  to  understand  the  reasons  for  the  robot's  actions  disrupted 
the  creation  of  common  ground.  Also,  many  fear  that  providing  more  and  more  autonomy  to 
armed  robots  could  result  in  collateral  damage  or  fratricide  (Singer,  2009). 

In  a  recent  reconnaissance  experiment  on  progressive  autonomy  in  robots  (Pettitt  et  al.,  2010), 
the  participants  agreed  that  mapping  a  room,  driving  the  robot,  and  responding  to  queries  from 
the  white  cell  in  the  teleoperation  mode  created  task  demands  that  were  too  difficult.  The 
Soldiers  rated  teleoperation  as  creating  a  higher  cognitive  and  overall  workload  and  higher  stress 
(frustration)  than  semi-  and  fully  autonomous  operation.  While  Dixon  and  Wickens  (2003) 
suggested  that  automation  would  relieve  cognitive  overload,  offloading  some  of  the  tasks  by  use 
of  the  autonomous  and  semi-autonomous  modes  relieved  more  than  just  the  cognitive  load  and 
made  the  performance  of  the  tasks  much  more  manageable.  It  was  clear  that  the  operators’ 
mental  models  of  the  environment,  based  upon  viewing  it  through  a  robotic  driving  camera,  were 
fairly  inaccurate  because  map  accuracy  in  the  teleoperation  mode  was  much  poorer  than  in  the 
autonomous  and  semi-autonomous  modes.  This  supports  the  findings  of  Fong  et.  al  (2003)  who 
indicated  that  operators  using  teleoperation  have  difficulty  building  mental  models  of  remote 
environments. 
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In  this  study,  we  again  examine  three  levels  of  automation:  (1)  teleoperation,  (2)  semi-autonomy 
(teleoperation  with  obstacle  avoidance)  and,  (3)  autonomy  (reflexive  waypoint  navigation  with 
obstacle  avoidance)  for  a  new  task,  intelligence,  surveillance,  and  reconnaissance  (ISR)  of  an 
area.  Once  again,  our  hypothesis  is  that  full  autonomy  will  be  the  most  effective  level  of 
autonomy  for  this  mission  when  the  robotic  operator  is  fully  engaged  in  a  high  cognitive  load 
activity. 

1.2  Objective 

The  goal  of  this  research  was  to  examine  the  effectiveness  of  different  levels  of  automation  on 
robot  control  in  ISR  missions  when  the  robotic  operator  is  fully  engaged  in  a  high  cognitive  load 
activity. 

1.3  Overview  Experiment 

This  study  was  a  cooperative  research  effort  between  the  U.S.  Anny  Research  Laboratory 
(ARL)-Human  Research  and  Engineering  Directorate  (HRED)  and  Space  and  Naval  Warfare 
Systems  Center,  San  Diego,  (SSC  Pacific).  The  study  was  an  investigation  of  the  effects  of 
progressive  levels  of  autonomy  on  ISR  robotic  navigation  tasks.  It  took  place  at  Fort  Benning, 
GA,  and  27  Soldiers  from  the  Officer  Candidate  School  (OCS)  participated  in  the  study.  After 
training  on  the  operation  of  the  robotic  system,  each  participant  navigated  the  robot  along  a  route 
littered  with  obstacles  while  looking  for  unexploded  ordinance  (UXO)  using  three  different 
levels  of  robotic  automation  (teleoperated,  semi-autonomous,  and  autonomous).  During  the 
exercise,  Soldiers  responded  to  requests  for  information  regarding  situation  and  mission 
awareness.  The  hazards  and  the  automation  level  were  counterbalanced  to  control  for  the  effect 
of  learning.  Automation  level  and  usability  were  evaluated  based  on  objective  performance  data, 
data  collector  observations,  and  Soldier  questionnaires. 


2.  Method 


2.1  Participants 

Twenty-seven  Soldiers  from  the  Fort  Benning  OCS  participated  in  the  assessment.  The  OCS 
participants  included  Soldiers  with  prior  enlisted  service  with  a  variety  of  backgrounds  and 
experience  levels,  as  well  as  those  just  coming  into  the  Army  from  college. 

2.1.1  Pretest  Orientation 

The  Soldiers  were  given  an  orientation  on  the  purpose  of  the  study  and  what  their  participation 
would  involve.  They  were  briefed  on  the  objectives,  procedures,  and  the  robotic  system.  They 
were  also  told  how  the  results  would  be  used  and  the  benefits  the  military  could  expect  from  this 
investigation.  Any  questions  the  subjects  had  regarding  the  study  were  answered. 


3 


2.2  Apparatus  and  Instruments 
2.2.1  SSC  Pacific  Robot 


The  SSC  Pacific  robot  (figure  1)  is  a  high  mobility  long-range  surveillance  robot  with  long  range 
communication,  extended  mission  endurance,  and  an  ISR  sensor  suite  appropriate  for  long  range 
ground  surveillance.  Autonomous  Solutions  Incorporated  Chaos  platfonn  hosts  the  sensor  suite. 
The  payload  contains  forward-looking  infrared  radar’s  (FLIR’s)  D100E  Pan/Tilt  Unit,  a  Small 
Tactical  Optical  Rifle  Mounted  (STORM)  Micro-laser  Rangefinder,  a  camera  suite  consisting  of 
a  thermal  100-inm  fixed  lens,  a  color  Day/Night  Electron  Multiplying  Charge  Coupled  Device 
(EMCCD)  Camera  300-mm  zoom  lens,  and  a  color  Day/Night  EMCCD  Camera  8 -mm  fixed 
lens.  A  GPS  is  used  for  navigation.  In  addition,  the  payload  contains  a  processor  that  runs  all 
the  autonomous  behaviors  onboard  the  robot  and  a  radio  to  communicate  with  the  operator 
control  unit. 


Figure  1.  SSC  Pacific  robot. 


2.2.2  Operator  Interface 

The  operator  interface  used  to  control  the  system  is  SSC  Pacific’s  Multi-Robot  Operator  Control 
Unit  (MOCU).  The  robot’s  location,  driven  path,  and  goal  points  are  overlaid  on  an  aerial 
image.  Real-time  video  from  the  robot  is  also  displayed.  The  controller  is  a  Microsoft  Xbox* 
360  Game  Controller  and  a  touch  screen  upon  which  the  control  functions  have  been  mapped. 
Various  button  controls  are  provided  to  send  drive  commands  to  the  robot,  as  well  as  to  change 
the  robot’s  behaviors. 


Microsoft  and  Microsoft  Xbox  are  registered  trademarks  of  Microsoft  Corporation  in  the  United  States  and  other  countries. 
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The  interface  allows  the  operator  to  achieve  a  type  of  sliding  autonomy  with  the  robot.  Levels  of 
autonomy  are  achieved  by  turning  on  and  off  robot  behaviors.  All  behaviors  were  turned  off 
during  teleoperation  so  that  the  operator  was  in  complete  control  of  all  robot  actions.  Semi¬ 
autonomy  (reflexive  teleoperation)  was  achieved  by  using  the  guarded  teleoperation  function 
available  on  the  robot.  This  function  assists  the  robot  operator  during  driving  by  automatically 
avoiding  obstacles.  With  reflexive  teleoperation,  the  operator  controlled  the  robot’s  movement 
through  the  joystick.  The  robot  was  able  to  override  the  user’s  commands  to  avoid  obstacles 
when  necessary.  Full  autonomy  was  achieved  by  turning  on  the  guarded  teleoperation  behavior 
and  autonomous  waypoint  navigation.  In  this  level  of  autonomy,  the  operator  set  the  robot’s 
desired  destination  and  intermediary  points  and  the  robot  traversed  from  point  to  point  utilizing 
GPS  data.  The  robot  can  deviate  from  these  paths  when  necessary  for  obstacle  avoidance.  Due 
to  intermittent  GPS  satellite  coverage  experienced  at  the  test  site,  a  representative  from  SSC 
Pacific  simulated  full  autonomy  by  manually  controlling  the  robot  during  the  fully  autonomous 
trials. 

The  MOCU  software  was  developed  independent  of  the  hardware  unit,  and  thus  can  be  used  on 
any  computer  with  a  Windows  XPf  or  newer  operating  system.  A  commercial  off-the-shelf 
(COTS)  grade  laptop  was  used  in  the  experiment  to  run  MOCU.  A  Microsoft  Xbox  360 
controller  was  used  to  provide  a  joystick  interface  for  manual  driving  of  the  robot. 

2.2.3  Robotic  Reconnaissance  Route 

The  reconnaissance  route  (figure  2)  was  on  a  200-m  length  of  roadway  littered  with  obstacles. 
The  obstacles  were  arranged  in  a  manner  that  allowed  the  operator  to  maneuver  the  robot  around 
them  without  having  to  leave  the  roadway.  Fifteen  UXOs  were  placed  along  the  route  at  various 
points.  The  start  point  and  end  point  were  marked  with  engineer  tape.  A  different  configuration 
of  obstacle  and  UXO  placement  was  used  for  each  iteration.  The  operator  was  located  at  a 
stationary  control  station  outside  the  line  of  sight  of  the  route. 


'  Windows  and  Windows  XP  are  registered  trademarks  of  Microsoft  Corporation  in  the  United  States  and  other  countries. 


5 


2.2.4  The  National  Aeronautics  and  Space  Administration-Task  Load  Index  (NASA- TLX) 

The  NASA-TLX  requires  the  user  to  rate  the  workload  of  a  device  on  a  number  of  different 
scales  and  to  assign  an  importance  weight  to  each  scale.  The  scores  on  the  workload  scales 
(mental,  physical,  temporal,  perfonnance,  effort,  and  frustration)  can  be  combined  in  an  overall 
workload  score  (Hart  and  Staveland,  1988). 

2.2.5  Questionnaires 

The  questionnaires  were  designed  to  elicit  Soldiers’  opinions  about  their  performance  and 
experiences  with  each  of  the  controller  systems.  The  questionnaires  asked  the  Soldiers  to  rate 
the  devices  on  a  7-point  semantic  differential  scale  ranging  from  “extremely  bad/difficult”  to 
“extremely  good/easy.”  Questionnaires  were  administered  to  each  Soldier  at  the  end  of  each 
iteration  (with  each  type  of  controller)  and  at  the  end  of  the  experiment.  Questionnaires  were 
also  used  to  gather  information  concerning  the  Soldiers’  demographic  data,  robotic  experience, 
and  physical  characteristics  that  might  affect  their  ability  to  operate  the  robot. 

2.3  Procedures 

2.3.1  Soldier  Orientation 

In  the  experiment  Soldiers  reported  in  groups  of  six  for  one  day  each,  from  0800  to  1700  hours 
daily.  Upon  arrival,  they  received  a  roster  number  used  to  identify  them  throughout  the 
evaluation.  The  Soldiers  completed  an  infonned  consent  fonn,  medical  status  form,  and  a 
demographics  questionnaire.  They  were  given  an  oral  operations  order  that  explained  the  robotic 
mission  that  they  would  undertake  during  the  experiment.  The  training  and  robotic  courses  were 
also  explained,  and  any  questions  the  Soldiers  had  concerning  the  experiment  were  answered. 
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2.3.2  Training 


A  representative  from  SSC  Pacific  trained  the  Soldiers  on  the  use  of  the  ISR  robot.  Soldiers 
practiced  teleoperating  the  robot  on  the  same  routes  used  during  the  experiment  to  help  mitigate 
learning  effects.  They  were  trained  on  each  control  condition  just  before  executing  the  course 
with  that  condition.  Soldiers  were  considered  trained  once  they  were  able  to  complete  the  route 
without  assistance.  The  average  training  time  required  was  20  min.  Questionnaires  concerning 
the  amount  of  practice  time  given,  the  level  of  detail  presented,  and  the  adequacy  of  training  aids 
were  administered  at  the  completion  of  training. 

2.3.3  Reconnaissance  Route  Iterations 

Soldiers  completed  three  iterations  of  the  robotic  driving  course,  once  using  each  level  of 
autonomy  (table  1).  Each  iteration  with  its  associated  level  of  autonomy  consisted  of  three  trials. 
The  first  was  a  training  practice  trial,  the  second  was  a  record  trial,  and  the  third  was  a  record 
trial  with  secondary  tasks  included.  Soldiers  were  given  an  operations  briefing  that  explained  the 
robotic  mission.  During  the  briefing,  it  was  explained  that  their  primary  task  was  to  negotiate  the 
route  as  quickly  as  possible  without  running  into  any  of  the  obstacles  and  without  leaving  the 
roadway.  Soldiers  were  also  instructed  to  report  any  UXO  they  observed  while  traversing  the 
route. 


Table  1.  Order  of  treatments  and  lanes. 


Roster 

Iteration  1 

Iteration  2 

Iteration  3 

1,  7,  13,  19,25 

A 

S 

T 

2,  8,  14,20,26 

A 

T 

S 

3,9,  15,21,27 

S 

A 

T 

4,  10,  16,  22,28 

S 

T 

A 

5,  11, 17,23,29 

T 

A 

S 

6,  12,  18,24,30 

T 

S 

A 

The  Soldiers’  secondary  tasks,  conducted  in  the  third  trial,  included  reporting  the  following  when 
requested  by  the  data  collector:  current  course  time  using  a  stop  watch,  the  correct  number  codes 
using  a  matrix  (table  2)  when  rows  and  columns  were  given,  and  the  activities  and  location  of  a 
person  of  interest.  The  person  of  interest  was  located  in  the  vicinity  of  the  operator  and 
perfonned  various  scripted  activities  during  the  course  of  the  reconnaissance  (table  3).  Soldiers 
were  required  to  report  their  responses  to  the  secondary  tasks  via  radio  to  the  data  collector.  The 
secondary  tasks  served  as  an  additional  physical  and  cognitive  load.  Secondary  task  questions 
were  asked  at  20-second  (s)  intervals  throughout  the  reconnaissance. 
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Table  2.  Secondary  task  code  matrix. 


A 

R 

B 

Q 

C 

P 

D 

O 

E 

N 

F 

M 

G 

L 

H 

K 

I 

J 

1 

2 

1 

0 

3 

4 

6 

7 

8 

9 

2 

1 

0 

3 

4 

6 

7 

8 

9 

18 

4 

3 

5 

6 

8 

12 

8 

9 

10 

4 

3 

5 

6 

8 

12 

8 

9 

10 

3 

6 

5 

10 

9 

12 

18 

9 

10 

11 

6 

5 

10 

9 

12 

18 

9 

10 

11 

16 

8 

7 

15 

12 

16 

24 

10 

11 

12 

8 

7 

15 

12 

16 

24 

10 

11 

12 

5 

10 

9 

20 

15 

20 

30 

11 

12 

13 

10 

9 

20 

15 

20 

30 

11 

12 

13 

14 

12 

11 

25 

18 

24 

36 

12 

13 

14 

12 

11 

25 

18 

24 

36 

12 

13 

14 

7 

14 

13 

30 

21 

28 

42 

13 

14 

15 

14 

13 

30 

21 

28 

42 

13 

14 

15 

12 

16 

15 

35 

24 

32 

48 

14 

15 

16 

16 

15 

35 

24 

32 

48 

14 

15 

16 

9 

18 

17 

40 

27 

36 

54 

15 

16 

17 

18 

17 

40 

27 

36 

54 

15 

16 

17 

10 

20 

19 

45 

30 

40 

60 

16 

17 

18 

20 

19 

45 

30 

40 

60 

16 

17 

18 

11 

22 

21 

50 

33 

44 

66 

17 

18 

19 

22 

21 

50 

33 

44 

66 

17 

18 

19 

8 

24 

23 

55 

36 

48 

72 

18 

19 

20 

24 

23 

55 

36 

48 

72 

18 

19 

20 

13 

26 

25 

60 

39 

52 

78 

19 

20 

21 

26 

25 

60 

39 

52 

78 

19 

20 

21 

6 

28 

27 

65 

42 

56 

84 

20 

21 

22 

28 

27 

65 

42 

56 

84 

20 

21 

22 

15 

30 

29 

70 

45 

60 

90 

21 

22 

23 

30 

29 

70 

45 

60 

90 

21 

22 

23 

4 

32 

31 

75 

48 

64 

96 

22 

23 

24 

32 

31 

75 

48 

64 

96 

22 

23 

24 

17 

34 

33 

80 

51 

68 

100 

23 

24 

25 

34 

33 

80 

51 

68 

100 

23 

24 

25 

2 

36 

35 

85 

54 

72 

106 

24 

25 

26 

36 

35 

85 

54 

72 

106 

24 

25 

26 

Table  3.  Secondary  task  script. 


Iteration  #1  -  Secondary  Tasks  -  20-s  intervals 

1 

Course  Time 

2 

Matrix  B-5 

3 

Person  of  Interest  (behind  operator  with  M-4) 

4 

Matrix  P-6 

5 

Course  Time 

6 

Person  of  Interest  (left  side  behind  operator  with  shovel) 

7 

Course  Time 

8 

Matrix  G-2 

9 

Person  of  Interest  (right  side  behind  operator  with  radio) 

10 

Course  Time 

11 

Matrix  E-10 

12 

Matrix  H-9 

13 

Person  of  Interest  (behind  operator  writing) 

14 

Course  Time 

15 

Person  of  Interest  (walking  left  to  right) 

Iteration  #2  -  Secondary  Tasks  -  20-s  intervals 

1 

Matrix  J-2 

2 

Person  of  Interest  (behind  operator  in  prone  position) 

3 

Course  Time 

4 

Course  Time 

5 

Matrix  E-8 

6 

Person  of  Interest  (behind  operator  with  engineer  tape) 

7 

Person  of  interest  (left  behind  operator  with  M-4) 

8 

Course  Time 

9 

Matrix  M-13 

8 


Table  3.  Secondary  task  script  (continued). 


Iteration  #1  -  Secondary  Tasks  -  20-s  intervals 

10 

Matrix  L-ll 

11 

Course  Time 

12 

Person  of  interest  (right  behind  operator  talking  on  radio) 

13 

Matrix  Q-15 

14 

Course  Time 

15 

Person  of  Interest  (standing  with  shovel) 

Iteration  #3  -  Secondary  Tasks  -  20-s  intervals 

1 

Person  of  interest  directly  behind  operator  digging 

2 

Course  Time 

3 

Matrix  1-17 

4 

Course  Time 

5 

Matrix  F-10 

6 

Person  of  Interest  (behind  operator  with  radio) 

7 

Person  of  interest  (left  behind  operator  in  prone  position) 

8 

Course  Time 

9 

Matrix  C-9 

10 

Person  of  interest  right  behind  operator  with  engineer 
tape 

11 

Course  Time 

12 

Person  of  interest  right  behind  operator  with  M-4 

13 

Matrix  D-15 

14 

Course  Time 

15 

Matrix  0-8 

A  data  collector  following  the  robot  recorded  course  completion  time,  number  of  driving  errors, 
and  number  of  times  off  course.  A  driving  error  was  recorded  when  the  robot  ran  into  an 
obstacle  or  over  UXO  and  an  off  course  error  was  recorded  if  either  track  of  the  robot  left  the 
roadway.  A  data  collector,  co-located  with  the  operator,  recorded  the  number  of  secondary  tasks 
perfonned  correctly  or  incorrectly  and  recorded  the  number  of  UXO  reported. 

Upon  completing  each  iteration,  the  Soldiers  were  given  a  questionnaire  designed  to  assess  their 
perfonnance  and  experiences  with  each  of  the  modes  of  operation.  The  participants  also 
completed  the  NASA-TLX. 

2.3.4  End  of  Experiment  Questionnaire 

After  completing  all  three  iterations,  the  Soldiers  completed  an  end-of-experiment  questionnaire 
that  compared  each  of  the  control  methods  on  a  number  of  characteristics.  They  also  completed 
questionnaires  concerning  the  infonnation  requirements  for  teleoperating  the  robot. 
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3.  Results 


3.1  Demographics  Results 

The  27  participants  were  OCS  candidates  assigned  to  the  OCS,  Fort  Benning,  GA.  The  mean 
age  of  the  Soldiers  was  27  years  (range  23-34),  and  the  average  time  in  service  was  29  months 
(range  2-130).  None  of  the  Soldiers  had  any  prior  military  experience  in  teleoperating  a  ground 
unmanned  robot.  Detailed  responses  to  the  demographics  questionnaire  are  available  in 
appendix  A. 

3.2  Training  Results 

Participants  stated  the  training  they  received  was  thorough  and  fully  prepared  them  to  perform 
the  tasks  required  to  conduct  the  robotic  reconnaissance  course.  Learning  to  operate  the  controls 
and  drive  the  robot  was  rated  as  being  very  easy  for  all  control  conditions.  Several  Soldiers 
indicated  that  the  hardest  task  to  leam  was  maneuvering  the  robot  around  objects  with  the 
obstacle  avoidance  system  turned  on.  Detailed  responses  to  the  training  questionnaire  are 
available  in  appendix  B. 

3.3  Robotic  Reconnaissance  Course  Results 

The  summary  statistics  for  course  completion  times,  UXOs  detected,  and  driving  errors  are 
shown  in  table  4. 


Table  4.  Means  and  (SDs),  course  time,  UXOs  detected  and  driving  errors. 


Without  secondary  tasks: 


Measure 

Teleoperation 

Semi-autonomous 

Autonomous 

Time  (min:  sec) 

2:43  (0:57) 

3:13  (1:11) 

2:37  (0:25) 

UXOs  detected 

9.52  (3.19) 

9.48  (3.48) 

9.96  (3.20) 

Errors 

2.19(2.29) 

1.26  (1.38) 

0 

With  secondary  tasks: 


Measure 

Teleoperation 

Semi-autonomous 

Autonomous 

Time  (min:  sec) 

3:04  (1:04) 

3:41  (1:13) 

2:38  (0:22) 

UXOs  detected 

9.07  (3.44) 

10.15  (3.34) 

9.22  (3.13) 

Errors 

2.44  (2.94) 

1.04  (0.98) 

0 
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The  course  completion  times  (figure  3)  were  analyzed  using  a  repeated  measures  analysis  of 
variance  (ANOVA).  The  main  effect  for  control  modality  [F(2,52)  =  10.5,  p  <  .00 1 , // 2 p  =  .287] 
was  statistically  significant,  as  was  the  main  effect  for  secondary  tasks,  F(l,26)  =  23.2, p  <  .001, 
r/  p=  .472.  The  control  x  secondary  task  interaction  was  also  statistically  significant,  F{ 2,52)  = 
5.84 ,p=  .005,  rj  p=  .183.  Post-hoc  paired  comparisons  were  conducted  using  the  Bonferroni 
correction  for  family-wise  error.  In  both  the  teleoperation  [?( 26)  =  3.06,/?  <  .05,  d  =  .38]  and 
semi-autonomous  control  conditions  [7( 26)  =  4.45,/?  <  .05,  d=  .55],  the  participants  completed 
the  course  significantly  faster  without  the  secondary  tasks.  In  contrast,  the  secondary  task  had  no 
significant  effect  on  mean  course  times  in  the  autonomous  control  condition,  t  <  1 .  The  mean 
times  for  the  semi-autonomous  condition  was  slower  than  mean  times  for  the  teleoperation 
condition  with  [7(26)  =  5.56,/?  <  .05,  d=  .69]  and  without  [7(26)  =  4.17,/?  <  .05,  d=  .52]  the 
secondary  tasks.  Similarly,  the  mean  times  for  the  semi-autonomous  condition  were  slower  than 
means  for  the  autonomous  condition  with  [7(26)  =  9. 1 8,  p  <  .05,  d  =  1.14]  and  without  [7(26)  = 
4.45,/?  <  .05,  d=  .55]  the  secondary  tasks. 


Figure  3.  Course  completion  times  (mirnsec). 

An  analysis  of  the  number  of  UXOs  detected  (figure  4)  yielded  no  significant  main  effect  for 
control  condition,  for  secondary  tasks,  or  for  the  control  x  secondary  task  interaction,  all  F’s  <  1. 
A  ceiling  effect  was  observed  as  Soldiers  found  most  all  of  the  UXOs. 
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Figure  4.  Mean  number  of  UXOs  detected. 

There  were  no  driving  errors  at  all  in  the  autonomous  control  condition  (see  figure  5).  An 
analysis  of  driving  errors  in  the  teleoperation  and  semi-autonomous  conditions  indicated  that 
there  was  a  significant  main  effect  for  control  condition  [F(I,  26)  =  5.57, p  =  .026,  rj  p  =  .176], 
with  fewer  driving  errors  in  the  semi-autonomous  conditions.  Neither  the  main  effect  for 
secondary  tasks  nor  the  interactions  were  statistically  significant,  bothF’s  <  1. 


Figure  5.  Mean  number  of  driving  errors. 
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Table  5  summarizes  performance  on  the  secondary  tasks. 
Table  5.  Summary  statistics,  secondary  task  performance. 


Secondary  tasks 

Teleoperation 

Semi-autonomous 

Autonomous 

%  correct 

96%  (8%) 

98%  (4%) 

97%  (9%) 

%  stop 

73% 

63% 

5% 

The  participants  responded  correctly  to  almost  all  of  the  secondary  task  requests  in  all  three 
control  conditions;  there  were  no  significant  differences  among  the  conditions:  F( 2,52)  <  1. 

There  was  a  statistically  significant  difference  among  the  control  conditions  in  the  proportion  of 
times  the  Soldier  stopped  operating  the  robot  in  order  to  respond  to  the  secondary  task  request: 
F{ 2,52)  =  84.7,  p<  .001,  rj  ~p  =  .765;  figure  6.  Follow-on  paired-sample  t-test  comparisons  were 
conducted  using  the  Holm’s  sequential  Bonferroni  correction  for  family-wise  error  rate.  As 
shown  in  table  6,  there  were  significantly  fewer  stoppages  in  the  autonomous  condition  than  in 
the  teleoperation  or  semi-autonomous  conditions.  The  percentage  of  stoppages  in  the  semi- 
autonomous  condition  and  the  teleoperation  condition  comparison  approached  significance  and 
the  effect  size  was  moderate. 

80% 

70% 

60% 

50% 

40% 

30% 

20% 

10% 

0% 

Teleoperation  Semi-autonomous  Autonomous 


Figure  6.  Percent  of  times  Soldier  stopped  robotic  operation  in  order  to  respond  to  secondary  tasks. 
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Table  6.  Follow-on  comparisons,  percentage  stop. 


Pair 

t 

df 

obtained  p 

required/; 

Cohen’s  d 

Tele  vs.  Semi 

1.86 

26 

0.075 

0.05 

0.31 

Tele  vs.  Auto 

12.02 

26 

<.001 

0.017 

3.15 

Semi  vs.  Auto 

9.47 

26 

<.002 

0.025 

2.31 

3.4  NASA-TLX  Results 

The  NASA-TLX  scale  means  and  total  workload  means  are  shown  in  table  7  and  figure  7. 

Table  7.  Means  and  (SDs),  NASA-TLX 


Scale 

Teleoperation 

Semi-autonomous 

Autonomous 

Mental 

53.1  (22.2) 

58.7(21.6) 

39.6  (20.6) 

Physical 

15.0(12.6) 

18.1  (16.3) 

11.1  (9.7) 

Temporal 

53.9  (20.8) 

52.2  (21.4) 

42.8  (21.2) 

Performance 

36.5  (21.5) 

42.8  (24.6) 

33.7  (22.8) 

Effort 

51.5  (24.2) 

52.2  (22.9) 

39.3  (21.1) 

Frustration 

38.0  (22.4) 

44.8  (28.6) 

31.1  (19.7) 

Total  workload 

48.5  (14.2) 

52.8  (18.1) 

39.0(16.6) 

Figure  7.  NASA-TLX  scale  means. 

The  repeated  measures  ANOVAs  are  summarized  in  table  8,  and  the  follow-on  paired 
comparisons  are  shown  in  table  9.  On  the  Mental  Scale,  both  teleoperation  and  semi- 
autonomous  control  received  higher  mean  workload  ratings  than  autonomous  control.  The  semi- 


14 


autonomous  control  was  rated  higher  than  autonomous  control  in  terms  of  Physical  and 
Temporal  demands.  Autonomous  control  was  rated  as  requiring  less  effort  than  teleoperation 
and  semi-autonomous  control.  The  semi-autonomous  control  was  rated  higher  than  autonomous 
control  in  terms  of  Frustration.  Finally,  autonomous  control  was  rated  as  being  lower  in  total 
workload  than  teleoperation  and  semi-autonomous  control. 

Table  8.  Repeated  measures  ANOVAs,  NASA-TLX. 


Scale 

df 

F 

P 

>I2p 

Mental 

2,52 

11.30 

<.001 

0.302 

Physical 

2,52 

4.94 

0.011 

0.160 

Temporal 

2,52 

3.55 

0.036 

0.120 

Performance 

2,52 

1.82 

0.172 

0.066 

Effort 

2,52 

5.79 

0.005 

0.182 

Frustration 

2,52 

3.62 

0.034 

0.122 

Total  workload 

2,52 

11.20 

<.001 

0.301 
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Table  9.  Follow-on  paired  comparisons,  NASA  TLX. 


Mental: 


Pair 

t 

df 

obtained  p 

required  p 

Cohen's  d 

Tele  vs.  Semi 

1.87 

26 

0.073 

0.050 

0.27 

Tele  vs.  Auto 

2.77 

26 

0.010* 

0.025 

0.66 

Semi  vs.  Auto 

4.41 

26 

<.001* 

0.017 

0.93 

Physical: 


Pair 

t 

df 

obtained  p 

required  p 

Cohen's  d 

Tele  vs.  Semi 

1.50 

26 

0.147 

0.050 

0.15 

Tele  vs.  Auto 

1.96 

26 

0.061 

0.025 

0.19 

Semi  vs.  Auto 

2.72 

26 

0.012* 

0.017 

0.34 

Temporal: 


Pair 

t 

df 

obtained  p 

required  p 

Cohen's  d 

Tele  vs.  Semi 

1.50 

26 

0.147 

0.050 

0.08 

Tele  vs.  Auto 

1.96 

26 

0.061 

0.025 

0.54 

Semi  vs.  Auto 

2.72 

26 

0.012* 

0.017 

0.46 

Effort: 


Pair 

t 

df 

obtained  p 

required  p 

Cohen's  d 

Tele  vs.  Semi 

>  1 

26 

0.860 

0.050 

0.03 

Tele  vs.  Auto 

3.04 

26 

0.005* 

0.017 

0.59 

Semi  vs.  Auto 

2.80 

26 

0.009* 

0.025 

0.63 

Frustration: 


Pair 

t 

df 

obtained  p 

required  p 

Cohen's  d 

Tele  vs.  Semi 

1.32 

26 

0.199 

0.050 

0.33 

Tele  vs.  Auto 

1.44 

26 

0.161 

0.025 

0.34 

Semi  vs.  Auto 

2.58 

26 

0.016* 

0.017 

0.67 

Total  workload: 


Pair 

t 

df 

obtained  p 

required  p 

Cohen's  d 

Tele  vs.  Semi 

1.53 

26 

0.137 

0.050 

0.21 

Tele  vs.  Auto 

3.04 

26 

0.005* 

0.025 

0.46 

Semi  vs.  Auto 

4.57 

26 

<.001* 

0.017 

0.67 

*  p  <  .05,  2 -tailed 
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3.5  Questionnaire  Results 


Upon  completion  of  each  iteration,  Soldiers  were  asked  to  rate  their  ability  to  perform 
navigation/maneuver  tasks  on  a  7-point  semantic  differential  scale  ranging  from  “extremely 
difficult”  to  “extremely  easy.”  Figure  8  shows  Soldier  task  ratings  for  each  level  of  autonomy. 


Overall  mission  performance 
Ability  to  multitask 
Navigateto  waypoints 
Maintain  SA 
Finish  the  course  quickly 
Use  the  map  to  plan  route 
Avoid  obstacles 


0  1  2  3  4  5  6 


U  Autonomous 
H  Semi-autonomous 
ii  Teleop 


Figure  8.  Soldier  task  ratings  for  each  level  of  autonomy. 

The  Soldiers  preferred  full  autonomy  for  all  the  navigation/maneuver  tasks  required  to  complete 
the  reconnaissance  course.  Once  the  operator  set  the  robot’s  desired  destination  and 
intermediary  points,  the  robot  traversed  from  point  to  point  without  further  input  from  the 
operator.  The  hands-free  nature  of  autonomous  control  enabled  the  Soldiers  to  focus  more 
attention  on  the  secondary  tasks.  The  disadvantage  of  autonomous  control  identified  by  the 
Soldiers  was  the  inability  to  thoroughly  search  the  route  for  UXOs.  In  the  autonomous  mode,  the 
robot  continuously  moves  and  automatically  selects  routes  around  obstacles.  Several  Soldiers 
stated  that  they  would  prefer  to  stop  the  robot  when  distracted  by  a  secondary  task  so  they  would 
not  miss  something  while  they  were  looking  away  from  the  MOCU. 

The  semi-autonomous  mode  was  preferred  over  the  teleoperated  mode  for  multitasking, 
maintaining  SA,  avoiding  obstacles,  and  using  the  map  to  plan  routes.  Soldiers  stated  that  when 
operating  in  the  semi-autonomous  mode,  they  could  confidently  move  the  robot  faster  without 
worrying  about  hitting  obstacles  and  could  perform  secondary  tasks  easier  than  when  fully 
teleoperating  the  robot.  The  disadvantage  of  the  semi -autonomous  mode  identified  by  the 
Soldiers  was  the  unpredictable  nature  of  the  guarded  teleoperation  behavior.  When  approaching 
an  obstacle,  the  robot  would  automatically  turn  away  from  it  to  avoid  contact.  If  the  operator 
intended  to  go  around  an  obstacle  to  the  right  or  left  side,  the  robot  often  turned  away  in  the 
opposite  direction  causing  the  operator  to  take  a  different  route  than  intended  or  having  to  back 
the  robot  up  and  re-approach  it  from  a  different  angle.  Several  Soldiers  stated  that  the 
unpredictable  nature  of  the  guarded  teleoperation  was  frustrating  and  caused  them  to  feel  like 
they  were  competing  for  control  of  the  robot. 
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In  the  teloperated  mode,  Soldiers  stated  they  felt  they  had  more  control  of  the  robot’s  speed  and 
direction  and  they  were  able  to  perform  a  more  thorough  reconnaissance  for  UXOs.  Several 
disadvantages  of  teleoperation  were  identified  when  compared  to  the  other  modes.  Most 
Soldiers  agreed  that  the  teleoperation  mode  was  more  mentally  and  physically  demanding  and 
required  more  focus  than  the  other  modes,  therefore  making  multi-tasking  and  secondary  task 
perfonnance  more  difficult.  They  also  stated  that  teleoperation  requires  more  skill  with  the 
controller  to  avoid  hitting  obstacles  and  UXOs. 

Detailed  responses  to  the  post-iteration  and  end-of-experiment  questionnaires  are  available  in 
appendices  C  and  D. 


4.  Discussion  and  Recommendations 


Because  the  secondary  tasks  were  fairly  simple  and  the  operators  in  many  cases  stopped  the 
forward  progress  of  the  robot  to  perform  the  secondary  tasks,  the  results  exhibited  a  ceiling  effect 
and  it  was  impossible  to  compare  the  operators’  situation  awareness  when  using  the  three  levels 
of  autonomy  to  see  if  findings  were  similar  to  those  of  Chen  and  Joyner,  2006;  Dixon  et  ah, 

2003;  and  Luck  et  ah,  2006. 

The  guarded  teleoperation  capability  of  the  robot  created  mixed  results.  Guarded  teleoperation 
theoretically  is  useful  in  cluttered  environments,  as  it  can  be  used  to  prevent  the  operator  from 
driving  into  objects  easily  detected  by  the  robot,  freeing  them  to  concentrate  on  higher  level 
decision  making.  However,  Soldiers  stated  that  the  hardest  thing  to  learn  was  how  to  maneuver 
the  robot  around  objects  when  the  obstacle  avoidance  system  was  being  used.  Although  the 
obstacle  avoidance  system  appeared  to  adversely  impact  course  completion  times,  it  did  reduce 
the  number  of  driving  errors  over  those  with  teleoperation  and  it  also  reduced  the  percentage  of 
times  the  operators  had  to  stop  driving  in  order  to  perform  the  secondary  task  over  the 
teleoperation  percentage.  This  capability  needs  to  be  refined  so  Soldiers  can  choose  (if  they 
desire)  how  to  most  efficiently  maneuver  around  the  detected  objects  rather  than  having  to  rely 
on  what  they  feel  is  chance  concerning  which  direction  the  robot  will  move  after  detection  of  an 
object. 

Overall,  the  Soldiers  preferred  full  autonomy  and  secondary  tasks  had  little  effect  on  the  robotic 
driving  tasks  in  this  mode.  However,  Soldiers  did  not  feel  that  they  were  as  thorough  when 
looking  for  UXOs  because  they  had  no  control  on  the  robot  behavior  and  could  not  pan  the  area. 
Providing  the  capability  to  pan  the  camera  to  search  for  UXOs  would  allow  more  efficient  and 
effective  search  performance.  Perfonnance  using  semi-autonomous  control  was  mixed. 

Operators  were  able  to  perform  secondary  tasks  without  stopping  the  robot  as  much  as  they  had 
to  when  teleoperating  it  and  they  had  fewer  driving  enors  than  they  did  with  teleoperation. 
However,  course  times  were  slower  with  the  semi-autonomous  mode,  probably  because  the 
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guarded  teleoperation  capability  made  it  difficult  to  efficiently  maneuver  around  obstacles. 
Teleoperation  was  the  least  preferred  mode;  however,  all  primary  and  secondary  tasks  could  be 
performed  efficiently  in  this  mode  and  Soldiers  felt  that  this  mode  gave  them  more  control  to 
search  efficiently  for  UXOs. 


5.  Conclusions 


Operating  the  robot  in  the  autonomous  mode  was  the  best  condition  for  the  tasks  required  by  this 
experimental  protocol.  Secondary  task  requirements  had  little  or  no  effect  on  primary  task 
perfonnance  in  this  mode  and  the  Soldiers  preferred  this  mode  of  operation  over  the  semi- 
autonomous  and  teleoperation  modes. 
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Appendix  A.  Demographics 


This  appendix  appears  in  its  original  form,  without  editorial  change. 
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SAMPLE  SIZE  =  27 


Sex 

MOS 

Rank 

Duty  Position 

Male  -20 

09S-  18 

E-4-14 

OCS  -17 

Female  -  7 

19K-  1 

E-5  -  5 

SGL  -  1 

35F  -  2 

E-6  -  3 

Tank  Cdr-  1 

36B-  1 

E-7  -  1 

NR  -8 

88M-  1 

NR-  4 

92A-  1 

NR-  3 

With  which  hand  do  you  most  often  write?  23  right  4_  left 

With  which  hand  do  you  most  often  fire  a  weapon?  25  right  _2_  left 

Do  you  wear  prescription  lenses?  8  No  19  Yes 

If  Yes,  which  do  you  most  often  wear?  _5_  Glasses  _3_  Contacts 

Do  you  wear  glasses  or  contacts  while  firing  a  weapon?  _6_  glasses  2  contacts 

Which  is  your  dominant  eye?  24  Right  _3_  Left 

What  is  the  highest  level  of  education  you  have  achieved? 

0  GED  _0_  High  School  _0_  Some  college  _0_  Two-year  college  degree 
21  Bachelor’s  degree  _6_  Master’s  degree 
Where  do  you  currently  use  a  computer? 

18  Home  14  Work  1  Library  0  Do  not  use 
Other:  7  Barracks 


Mean 

SD 

Range 

Your  age: 

27  yrs 

2.8  yrs 

23-34 

Your  height: 

69  in 

4.1  in 

60-75 

Your  weight: 

167  lbs 

32.2  lbs 

110-240 

How  long  have  you  served  in  the  military? 

29  months 

37.3 

2  -  130 

24 


No  Yes 


Have  you  been  a  fire  team  leader? 

25 

2 

Have  you  been  a  squad  leader? 

21 

6 

Have  you  been  deployed  overseas? 

18 

9 

Have  you  been  deployed  in  a  combat  area? 

20 

7 

14.  How  often  do  you  do/use  the  following? 


Number  of  Responses 

Daily 

Weekly 

Monthly 

Once  every 
few  months 

Rarely 

Never 

Mouse 

0 

2 

0 

3 

2 

20 

Joystick 

5 

14 

4 

1 

2 

1 

Touch  screen 

0 

3 

0 

1 

0 

23 

Software  with  icons 

0 

0 

0 

0 

2 

25 

Pull-down  menu  * 

0 

0 

0 

0 

3 

24 

Graphics/drawing  features 
in  software  packages* 

3 

8 

1 

4 

3 

8 

Email 

1 

0 

0 

0 

1 

25 

Operate  a  radio-controlled 
vehicle 

14 

11 

2 

0 

0 

0 

Play  first  person  shooter 
computer/video  games 

6 

4 

9 

1 

6 

1 

Use  an  X-box  controller 

7 

4 

6 

4 

5 

1 

*  One  ‘no  response’ 


15.  Using  the  scale  below,  please  rate  your  skill  level  for  each  of  the  following  activities? 


None 

Beginner 

Intermediate 

Expert 

Operating  ground  unmanned  vehicles 

23 

3 

0 

0 

Operating  aerial  unmanned  vehicles 

25 

2 

0 

0 

Target  detection  and  identification 

12 

9 

6 

0 

Playing  commercial  video  games 

3 

11 

5 

8 

Training  with  Army  video  simulations 

7 

15 

5 

0 

16.  How  successful  do  you  think  you  will  be  at  robotic  driving? 


Number  of  Responses 

Very  successful 

4 

Successful 

11 

Neutral 

12 

Unsuccessful 

0 

Very  unsuccessful 

0 

25 


Intentionally  left  blank. 
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Appendix  B.  Training 


This  appendix  appears  in  its  original  form,  without  editorial  change. 
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SAMPLE  SIZE  =  27 


1 .  Using  the  scale  below,  please  rate  the  training  that  you  received  in  the  following 
areas: 


1  2  3  4  5  6  7 

Extremely  difficult  Very  difficult  Difficult  Neutral  Easy  Very  easy  Extremely  easy 


TRAINING  FEATURES 

MEAN  RESPONSE 

Completeness  of  introductory  training 

6.22 

Comprehension  of  the  overall  concept  of  the  robot 

6.33 

Training  on  operating  the  robot  ( Teleoperation ) 

6.33 

Training  on  operating  the  robot  ( Semi-autonomous ) 

6.07 

Training  on  operating  the  robot  ( Autonomous ) 

6.19 

Time  provided  to  practice  driving  the  robot  on  the  route 
reconnaissance  course 

6.44 

Explanation  of  how  to  complete  the  route  reconnaissance 
course  (independent  of  controller  type) 

6.41 

2.  What  were  the  easiest  and  hardest  training  tasks  to  learn? 


Comments  No.  of  Responses 


Easiest 

Autonomous  operation.  6 

Use  of  controller.  1 

Controlling  the  robot  itself  with  the  Xbox  controller.  2 

Xbox  controller  because  of  past  experience  and  because  it  translates 
exactly.  1 

Controlling  robot  in  teleoperation.  4 

Spotting  items  of  interest.  1 

Driving/operating  the  robot.  5 

Ability  to  navigate  the  controller  for  the  robot.  1 

Driving  the  robot  in  semi-autonomous  mode.  1 

Moving  in  a  straight  line.  1 

Good  visuals.  1 

The  controls.  1 

Reading  the  computer  screen,  layout,  icons,  etc.  1 

Nice  speed.  1 

Controls  speed  well  so  you  can  maneuver  around  obstacles.  1 

Avoiding  obstacles.  1 

Plotting  the  point  on  the  map.  1 

Creating  the  route.  1 

Calling  out  lEDs.  1 
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Comments 


No.  of  Responses 


Hardest 

On  semi-autonomous.  4 

Semi-autonomous  was  probably  harder  to  learn  how  to  control,  due  to  the 
fact  it  won’t  always  go  in  the  direction  you  tell  it.  1 

Semi-autonomous  is  hardest  to  learn  because  it’s  difficult  to  predict  which 
way  the  robot  will  turn.  1 

Competing  with  the  semi-autonomous.  1 

Seeing  lEDs  with  the  course  resolution  (of  the  camera)  and  in  shadow 
driving  in  semi-autonomous  mode.  1 

Identifying  the  lEDs  on  the  course  while  limited  by  robot  resolution  and 
jitter.  1 

Adapting  to  the  robot  movements  in  semi-autonomous  mode.  1 

Teleoperation.  5 

Using  the  OA  software.  1 

Controlling  robot  in  close  quarters.  1 

Operating  the  robot  while  trying  to  answer  the  matrix.  1 

How  to  drive  robot  and  not  hit  objectives.  1 

Performing  secondary  tasks  while  driving  robot.  1 

Ability  to  multi-task  while  controlling  the  robot  and  identifying  lEDs.  1 

Understanding  the  relation  between  the  location  of  the  camera  on  the  robot 
and  the  image  on  the  screen.  1 

Being  comfortable  with  the  controller.  1 

Adding  the  secondary  action  of  “person  of  interest.”  1 

Having  to  multi-task,  particularly  when  looking  for  the  person  of  interest, 
was  the  most  difficult  task.  1 

Programming  the  route,  though  it  wasn’t  very  difficult  once  explained.  1 

It  was  difficult  to  predict  which  way  it  would  turn  to  avoid  obstacles.  1 

Video  malfunctions/glitches.  1 

Sensitive  joysticks.  1 

Navigating  with  the  joystick.  1 

Spatial/depth  perception.  1 

Timing  of  reaction.  1 

Lag  on  the  screen.  1 

Training  comments 

Very  professional  and  well  done.  1 

Enjoyable!  I’m  excited  about  new  technology.  1 

Very  interesting.  1 

Very  straightforward.  1 

Easy  to  follow  along  with.  1 

Provided  me  a  good  understanding  of  how  to  control  the  robot  and  my 
mission.  1 

Easy  to  understand  the  system  and  learn  the  controls.  1 

Not  as  hard  as  I  thought  it  would  be;  I  believe  anyone  could  operate  the 
robot.  1 
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Comments 


No.  of  Responses 


My  first  time  operating  a  robot;  very  satisfied  and  now  interested  in  how 
this  actually  happens.  I  have  heard  robotics  would  be  the  future;  this  was 
definitely  worth  coming  to  train  for.  1 

Semi-autonomous  was  the  best  mix  of  positives  from  autonomous  and 
teleoperation.  1 

Teleoperation  was  easy  to  learn,  but  most  difficult  to  perform. 

Possibly  more  specific  instruction  on  how  OA  works  and  makes 
determinations  would  help  to  instruct  the  driver  how  to  work  with  other 
than  against  the  system.  1 
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Appendix  C.  Post  Iteration 


This  appendix  appears  in  its  original  form,  without  editorial  change. 


SAMPLE  SIZE  =  27 


A  (Teleoperation);  B  (Semi-autonomous);  C  (Autonomous) 

1 .  Using  the  scale  below,  please  rate  your  ability  to  perform  each  of  the  following  tasks 
based  on  your  experience  with  the  autonomy  level  that  you  just  used: 

1  2  3  4  5  6  7 

Extremely  difficult  Very  difficult  Difficult  Neutral  Easy  Very  easy  Extremely  easy 


DRIVING  TASKS 

MEAN 

1  RESPONSE 

A 

B 

C 

a.  Move  the  robot  in  the  correct  direction 

5.30 

4.41 

6.43 

b.  Avoid  obstacles 

4.44 

4.78 

6.30 

c.  Turn  quickly 

4.26 

3.96 

6.25 

d.  Turn  precisely 

4.04 

3.85 

6.15 

e.  Anticipate  whether  turn  radius  of  vehicle  will  allow  a  turn 

4.15 

4.19 

6.00 

f.  Stop  quickly 

5.59 

5.63 

5.72 

g.  Get  up  to  speed  quickly 

5.67 

5.30 

6.35 

h.  Adjust  to  the  effects  of  system  latency  (lag) 

5.11 

4.48 

5.73 

i.  Adjust  to  the  feel  of  the  control  system 

5.44 

4.59 

5.83 

j.  Use  the  map  to  plan  route  in  advance 

5.15 

5.26 

5.76 

k.  Drive  at  slowest  speeds 

5.52 

5.04 

5.53 

1.  Drive  at  medium  speeds 

5.46 

5.23 

5.53 

m.  Drive  at  fastest  speeds 

5.59 

5.37 

5.76 

n.  Finish  the  course  quickly 

4.85 

4.33 

6.16 

o.  Responsiveness  of  the  robot  to  your  commands 

5.12 

3.96 

6.21 

p.  Drive  a  straight  route 

5.50 

5.07 

6.45 

q.  Maintain  situation  awareness 

4.59 

4.89 

5.48 

r.  Navigate  to  the  next  waypoint  or  set  of  hash  lines 

5.27 

5.08 

6.25 

s.  Ability  to  multitask  (operate/monitor  robot  and  communicate 
on  the  radio) 

4.22 

4.63 

5.44 

t.  Overall  ability  to  perform  this  mission 

5.07 

4.74 

5.63 

2.  Please  comment  on  every  task  above  that  gave  you  difficulty  and  try  to  pinpoint  the 
specific  problem  the  system  gave  you. 

Comments  No.  of  Responses 

Teleoperation 

All  tasks  were  fairly  simple  to  accomplish.  1 

Lease  difficult  of  the  three  because  it  allowed  for  the  quickest  navigation  of 
the  course.  1 
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Comments  No.  of  Responses 


I  am  not  a  gunner  and  did  not  have  practice  with  the  control  function, 
but  I  assume  it  would  be  much  easier  with  a  little  more  practice.  I  also 
have  pretty  bad  depth  perception  but  accustomed  to  the  robot  quickly.  1 
Multi-tasking.  2 

Multi-tasking  while  controlling  the  robot  was  a  challenge.  I  didn’t  feel 
comfortable  moving  the  robot  forward  while  performing  other  tasks.  1 
Multi-task  performing  the  many  courses  via  hand  controls,  video 
screen,  as  well  as  communicating  via  radio  was  difficult.  1 

Without  the  sensors,  I  tend  to  run  into  obstacles  often.  1 

If  comms  is  direct,  I  feel  that  the  comms  should  always  be  open  and 
not  require  an  over/out  (pushing  a  button).  1 

Avoiding  obstacles,  as  the  crosshair  in  the  view  is  well  off-centered.  1 
I  had  some  trouble  judging  turns  around  obstacles  using  the  computer 
screen.  1 

Much  more  difficult  than  with  obstacle  avoidance  (OA).  Unsure  of 
robot  in  regards  to  obstacles.  I  thought  I  might  have  hit  some  after 
passing  them  on  the  camera  view.  1 

Not  hitting  the  objects  in  the  course.  The  size  of  the  robot  as  I  drove 
through  the  obstacles.  1 

Navigating  quickly  around  obstacles  without  touching  them.  This  is 
likely  the  result  of  getting  used  to  the  joystick.  1 

Being  unfamiliar  with  the  sizes  of  objects  on  the  course  made  it  difficult 
to  maneuver  and  choose  the  best  route  around  obstacles.  1 

The  robot  ran  into  two  obstacles.  Some  areas  between  obstacles 
seemed  too  tight  of  a  space.  This  could  have  also  been  due  to  the 
fact  that  I  underestimated  the  size  of  the  robot.  1 

Over  steer  and  offset  camera  resulted  in  some  obstacle  collisions.  1 

When  turning  the  vehicle,  I  tended  to  over-correct.  1 

There  was  some  difficulty  with  seeing  smaller  objects  due  to  picture 
quality  and  lag;  picture  quality  greatly  reduced  depending  on  the  sun.  1 
Video  lag  was  a  problem.  1 

Low  screen  resolution  made  it  hard  to  spot  lEDs,  especially  in  the 
shadows.  2 

It  was  difficult  to  spot  lEDs  which  did  not  have  a  shape  in  contrast  to 
the  background.  1 

Abruptly  stopping  the  robot  to  do  my  tasks.  1 

Takes  more  focus.  1 

The  matrix  was  the  most  difficult;  mostly  because  I  had  to  focus  on  it 
the  most.  The  other  objectives  allowed  me  to  keep  my  finger  on  the 
control  while  I  performed  a  quick  task.  The  matrix  required  me  to 
literally  pinpoint  with  my  finger  the  answer  to  the  question.  1 

Still  have  to  stop  all  movement  to  find  items  on  the  matrix.  Very  time 
consuming.  1 
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Comments 


No.  of  Responses 


Semi-autonomous 

No  problems;  easier  than  manual  teleoperation  mode.  1 

The  change  in  location  when  the  screen  resumed  after  a  video  lag  was 
hard  to  adjust  to.  2 

Turning  robot  to  correct  direction  (wasn’t  moving  to  direction  on 
joystick).  1 

Maneuvering  the  vehicle  once  it  was  ‘stuck’  because  it  kept  trying  to 
autocorrect  at  times  I  didn’t  desire  it  to.  1 

Knowing  whether  it  was  you  or  the  robot  is  taking  control  of  the 
obstacle  avoidance  was  difficult.  Got  stuck  a  few  times.  2 

I  found  semi  much  harder  to  control  than  teleoperation.  When  the 
robot  corrected  itself,  I  found  the  angles  to  be  much  more  extreme 
than  had  I  done  it  myself.  I  found  myself  more  frustrated  when  it  took 
over  from  me.  1 

The  robot  did  not  respond  twice  when  I  attempted  to  make  a  sharp 
turn.  I  knew  I  was  past  an  obstacle  but  the  robot  did  not  respond  for  a 
few  seconds.  1 

When  the  system  used  the  OA  software,  it  made  the  course  slightly 
more  difficult  for  a  new  user.  1 

Compensating  for  the  OA  took  time  to  maneuver  better.  When  the 
robot  avoided  the  obstacles,  it  wasn’t  easy  to  readjust  smoothly.  1 

In  some  tight  situations,  I  had  to  fight  the  OA  when  my  course 
correction  called  for  in  place  rotation  to  either  side.  1 

When  using  the  collision  control  function,  it  seemed  to  prevent  my 
being  able  to  maneuver  the  robot  effectively.  1 

Moving  through  close  obstacles,  the  robot  continued  to  correct  as  to 
not  run  into  anything.  This  resulted  in  almost  no  movement  through 
narrow  corridors.  1 

Avoiding  obstacles,  stemming  from  adjusting  to  the  feel  of  the  control 
system,  making  turns  in  particular.  1 

Not  knowing  how  the  robot  would  avoid  obstacles  was  hard;  knowing  it 
would  veer  right  so  many  degrees  would  have  helped.  1 

In  tight  spots  it  looked  as  if  I  could  make  it  through,  but  the  controls 
wouldn’t  let  me  go.  2 

Turning  the  robot  quickly  was  difficult  if  the  robot  was  near  an  obstacle 
and  attempting  to  self-correct.  Turning  was,  in  general,  slightly 
clumsy.  1 

Getting  the  number  from  the  matrix  was  very  difficult  for  me.  1 

The  matrix  caused  me  to  completely  stop  what  I  was  doing;  was 
difficult  while  operating  robot.  2 

The  matrix  was  somewhat  difficult  to  read  because  of  the  letters  and 
numbers  being  out  of  order.  1 
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Comments 


No.  of  Responses 


The  task  that  requires  me  to  report  a  number  based  on  two 
coordinates  slowed  me  down  enough  to  simply  allow  the  robot  to  sit 
still  while  I  locate  the  two  coordinates.  1 

Feeling  in  control  with  direction,  had  difficulty  remaining  in  a  straight 
line.  1 

Multi-tasking  was  very  difficult  because  it  was  kind  of  hard  to  focus  on 
driving  while  handling  the  radio  and  looking  for  the  task.  1 

Autonomous 

Very  easy  to  use  autonomous  system.  Gives  you  the  ability  to  easily 
multi-task.  1 

The  movement  from  the  robot  is  much  faster  and  the  ability  to  dodge 
obstacle  was  effortless.  1 

Just  a  matter  of  getting  familiarized  with  the  screen  and  steps.  1 

Setting  points  wasn’t  very  precise  because  of  the  size  of  the  map  and 
the  touch  screen,  but  it  was  generally  fine.  1 

The  sensor  for  dropping  points  was  pretty  sensitive,  but  other  than 
that,  very  user  friendly.  1 

Where  I  was  exactly  on  the  map;  the  GPS  wasn’t  responding  well.  1 

I  forgot  to  switch  to  the  map  in  order  to  begin  plotting  route.  1 

The  control  system/map  would  be  something  that  I  would  need  more 
time  to  get  used  to.  1 

Unable  to  control  the  robot’s  speed,  but  did  not  find  this  to  be  an  issue.  1 
Only  difficulty  was  multi-tasking.  1 

When  trying  to  look  up  the  number  on  the  matrix,  it  was  difficult  to 
focus  on  the  screen.  1 

Identifying  the  person  of  interest  required  me  to  look  away  from  the 
screen  most  severely.  Scanning  the  matrix  which  is  out  of  the  order 
took  longer  than  a  sorted  matrix.  1 

The  matrix  not  being  in  sequence  reduced  my  ability  to  search  for 
lEDs  quickly.  1 

Difficult  identifying  the  person  of  interest  while  also  attempting  to 
identify  lEDs  along  the  route.  1 

When  I  had  to  take  my  attention  away  from  the  screen,  I  didn’t  have 
the  ability  to  stop  the  robot,  leaving  opportunity  for  me  to  miss  spotting 
lEDs.  1 

Screen  resolution  made  positive  IED  identification  difficult.  2 

Sometimes  I  couldn’t  tell  if  it  was  an  IED  or  other  debris.  1 

Identifying  objects.  1 

Harder  to  see  lEDs  in  this  mode.  1 

Poor  video  quality.  1 

Setting  the  course  for  the  robot  and  difficulties  seeing  color  images 
through  the  camera  created  the  most  difficulty.  1 
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Comments 


No.  of  Responses 


Frustrating  because  you  would  not  speed  up  or  slow  down  in  situations 
where  more  visual  information  was  needed  to  be  gathered.  1 

During  the  second  run,  I  did  not  know  how  to  connect/delete  two 
assigned  routes,  which  caused  my  final  time  to  be  longer  than 
intended.  I  ended  up  deleting  both  routes  and  starting  over  with  a 
new  route.  1 

3.  Please  check  any  of  the  following  conditions  that  you  may  have  experienced  during 
this  trial. 


CONDITION 

Number  of  Responses 

A 

B 

c 

Eyestrain 

4 

3 

4 

Tunnel  vision 

1 

1 

0 

Headaches 

0 

0 

0 

Motion  sickness 

0 

0 

1 

Nausea 

0 

0 

0 

Disorientation 

0 

0 

1 

Dizziness 

0 

0 

0 

Comments  No.  of  Responses 


Teleoperation 

Seeing  lEDs  in  the  shaded  portions  of  the  lane’s  edges.  1 

I  was  tense  throughout  the  obstacle  course.  1 

Semi-autonomous 

Controlling  the  robot  and  performing  tasks  at  the  same  time.  1 

Autonomous 

Performing  the  secondary  tasks  and  watching  the  screen  concurrently.  1 


4.  Using  the  scale  below,  what  is  your  overall  rating  of  the  controller  condition  that  you 
used  this  iteration? 


1  2 
Extremely  bad  Very  Bad 


3  4  5  6  7 

Bad  Neutral  Good  Very  Good  Extremely  Good 


MEAN  RESPONSE 

A 

B 

C 

5.52 

4.93 

5.81 
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Comments 


No.  of  Responses 


Teleoperation 

Overall  the  robot  was  easy  to  control,  especially  for  someone  completely 
new  to  the  system.  1 

I  think  the  controls  were  easy  to  use  and  interpret. 

Much  easier  in  terms  of  responsiveness.  1 

It  was  a  positive  experience  to  have  full  control  of  the  robot.  1 

Full  control  of  the  robot  allowed  me  to  divert  attention  from  driving  to  side 
and  allowed  me  to  turn  the  robot  to  assist  in  IED  identification.  1 

The  robot  was  easy  to  drive  and  extremely  responsive  to  commands  in  this 
mode.  1 

Easy  to  drive  because  it  did  not  jerk  left  or  right  because  of  small 
obstacles.  1 

Other  than  lag,  every  other  thing  about  the  system  was  excellent.  1 

Camera  lag  could  create  an  issue;  it  seems  to  interrupt  the  overall  mission 
of  moving  as  fast  as  possible  (no  matter  the  stopwatch),  just  in  general.  1 
I  think  given  more  time  and  familiarity  with  the  controls  and  the  robot  would 
eliminate  a  lot  of  the  initial  strain  and  frustration  levels.  1 

The  only  negatives  were  difficulties  due  to  picture  quality  and  lag.  1 

Semi-autonomous 

Overall,  I  felt  that  this  control  iteration  was  successful.  1 

Altogether,  using  the  robot  was  easy  to  learn  and  simple,  especially  in 
more  open  areas.  1 

The  controls  were  very  easy  to  use.  1 

This  control  condition  was  much  easier,  being  able  to  correct  the  robot, 
rather  than  correct  myself,  the  robot  and  movement.  The  robot  self 
corrected  and  allowed  me  to  maneuver  quickly  through  the  course.  1 

Less  stressful  than  having  full  control  of  the  robot,  as  there  was  no  worry 
about  hitting  obstacles.  1 

The  automatic  obstacle  detection  made  it  easy  to  drive  fast  with 
confidence.  2 

Visibility  of  obstacles  and  items  of  interest  was  good.  1 

The  system  indeed  has  a  good  mapping  capability  but  the  control  really 
could  use  some  fine  tuning  in  terms  of  turning  and  making  slow  to  medium 
speed  maneuvers.  1 

It  was  better  than  teleoperation,  but  I  would  prefer  autonomous  since  it  was 
difficult  moving  once  the  computer  took  over.  1 

Worked  great  except  through  narrow  passages.  1 

In  all  situations,  other  than  tight  corridors,  it  was  beyond  easy  to  control.  It 
took  some  adjustment  to  learn  to  work  with  and  anticipate  the  OA  course 
correction.  1 

OA  software  was  slightly  difficult  to  adjust  to.  1 

Most  difficulties  were  due  to  my  own  inexperience  with  the  system.  1 

I  liked  having  the  teleoperation  control  better  because  I  consistently  had 
control  over  the  robot  as  opposed  to  intermittent  control.  1 
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Comments 


No.  of  Responses 


I  found  the  teleoperation  easier  just  because  I  knew  how  the  robot  would 
move.  Every  time  the  robot  veered  because  of  an  obstacle,  I  had  to 
reorient  myself  to  the  surrounding  obstacles.  1 

I  really  struggled  with  maneuvering  around  obstacles,  and  felt  I  couldn’t 
drive  while  not  having  my  eyes  focused  on  the  screen.  1 

I  stunk  at  this  one!  1 

High  resolution  camera,  which  I  assume  will  be  used  anyway.  1 

Autonomous 

This  was  the  best!  Being  able  to  solely  focus  on  obstacles,  hazardous 
conditions  made  it  less  of  an  issue  because  full  focus  was  on  the  condition 
of  the  route  and  objects  throughout.  1 

Overall,  it  went  very  well.  With  some  practice,  I  could  see  myself  getting 
used  to  seeing  through  the  robot’s  eyes.  1 

Was  good  and  allowed  me  to  put  a  lot  more  focus  into  spotting  lEDs  better 
and  faster  than  in  the  teleoperation.  1 

Much  easier  than  the  teleoperation  and  leaves  time  for  secondary  tasks.  1 
By  far  the  easiest  and  least  stressful.  1 

Allowed  less  mental  demand  of  steering  the  robot.  1 

Allowed  me  to  complete  additional  tasks  with  easy  while  knowing  the 
primary  mission  would  be  accomplished  swiftly.  1 

Very  simple,  but  with  far  less  control  once  the  robot  started  moving.  1 

Although  somewhat  difficult,  it  was  interesting  to  have  other  factors  thrown 
into  the  mix  like  a  real  world  situation.  Autonomous  mode  made  the 
situation  a  little  more  difficult  because  of  the  inability  for  the  driver  to  stop 
the  vehicle.  1 

It  was  a  lot  easier  to  avoid  hitting  objects.  1 

Give  the  operator  the  power  to  pause  motion  to  accomplish  simultaneous 
tasks  while  operating  the  robot.  1 

I  was  better  able  to  focus  on  surroundings,  but  unable  to  view  terrain  and 
conditions  at  my  own  pace.  1 

Not  having  control  or  robot  made  me  less  observant.  1 

The  combination  of  control  and  observation  in  the  semi-autonomous  was 
better.  1 
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5.  Using  the  scale  below,  please  rate  the  degree  to  which  the  secondary  task  interfered 
with  your  ability  to  operate/monitor  the  robot. 

1  2  3  4 

Extreme  interference  Moderate  interference  Slight  interference  No  interference  at  all 


Scale  above-> 

Number  of  Responses 

Teleoperation 

Semi-aul 

:onomous 

Autonomous 

1 

2 

3 

4 

1 

2 

3 

4 

1 

2 

3 

4 

Report  numbers  from  matrix 

6 

12 

8 

1 

6 

11 

9 

1 

4 

6 

14 

3 

Report  time  on  course 

0 

6 

15 

6 

0 

1 

18 

8 

0 

0 

13 

14 

Report  on  person  of  interest 

4 

9 

12 

2 

3 

7 

13 

4 

1 

8 

11 

7 

Report  on  items  of  interest  * 

0 

8 

6 

10 

1 

3 

10 

10 

0 

6 

5 

13 

*  3  no  responses 


Comments  No.  of  Responses 


Teleoperation 

Overall  exercise  was  easy  to  navigate  and  adjust  to.  1 

It  was  easy  to  operate  the  radio  because  the  controller  only  needed  my  left 
hand  to  operate  it;  but  looking  around,  especially  at  the  POI,  was  difficult 
because  the  robot  was  still  moving  and  it  inhibited  my  ability  to  spot  lEDs 
because  I  was  focused  on  other  areas.  1 

Being  able  to  pause  at  will  using  the  controller  for  brief  intervals  eliminated 
the  pressure  of  multi-tasking  and  made  the  course  easy.  1 

Camera  jerk  was  predictable  and  controllable.  1 

Comms;  having  to  establish  communication  by  pushing  the  button  on  the 
mic  was  very  distracting,  as  well  as  turning  to  see  the  person  of  interest.  1 
The  secondary  tasks  were  not  difficult  or  distracting  except  when  they 
occurred  during  a  set  of  obstacles.  1 

Matrix  was  the  toughest  secondary  task.  2 

Having  to  search  for  the  letter  and  number  on  the  matrix  takes  at  least  a 
few  seconds.  2 

A  larger  font  size  and  familiarity  with  the  matrix  would  most  likely  be  much 
easier.  1 

Had  to  stop  what  I  was  doing  to  perform  secondary  task.  1 

When  maneuvering  around/avoiding  obstacles,  I  had  to  leave  off  of  driving 
to  complete  the  secondary  tasks.  1 

Turning  from  the  screen  completely  caused  a  delay  in  reorienting  direction 
and  screen  location.  2 


Whenever  a  task  called  for  me  to  take  my  eyes  off  of  the  screen,  it  caused 
an  extreme  interference  because  I  had  to  readjust  my  eyes  to  the  screen 
and  readjust  my  mental  focus.  The  matrix  was  the  most  extreme 
interference  because  I  had  to  locate  the  plots  and  come  up  with  a 
coordinate  as  opposed  to  just  observing  something  and  reporting  that 
back.  1 
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Comments 


No.  of  Responses 


Hard  to  turn  around  to  locate  and  describe  person  of  interest.  Indeed,  it 
took  away  my  ability  and  concentration  for  the  task  at  hand  to  operate  the 
robot  and  identify  lEDs.  1 

Semi-autonomous 

Not  as  much  as  the  last  time,  since  I  did  not  have  to  worry  about  running 
into  objects.  2 

Desirable  interference  with  robot  correcting  itself  in  front  of  objects.  1 

The  secondary  task  interference  made  the  exercise  somewhat  more 
difficult  paired  with  the  OA  software.  1 

I  assumed  the  letters  from  the  matrix  would  be  in  alphabetical  order.  That 
messed  me  up.  1 

The  matrix  is  evil.  1 

The  numbers  were  easily  the  hardest  task.  1 

Again,  plotting  the  numbers  from  the  matrix  takes  more  time  and  effort,  as 
opposed  to  just  reporting  objects,  persons  of  interest,  or  a  time  that  is 
easily  observable.  1 

The  numbers  slowed  me  down  by  having  me  to  switch  from  tunnel  vision  to 
side-to-side,  plus  up  and  down  eye  movements.  Time  and  person  of 
interest  was  easier  because  of  the  easy  recognition.  1 

If  and  when  to  report  lEDs  in  relation  to  waypoints.  I  saw  lEDs  but  due  to 
proximity  to  waypoints  I  could  not  tell  if  they  fell  under  the  former  or  latter 
points.  1 

Depending  on  whether  I  was  maneuvering  in  the  open  or  between 
obstacles,  my  ability  to  perform  the  secondary  tasks  was  either  easy  or 
slightly  difficult.  1 

Considering  the  system  setup  during  this  iteration,  it  was  necessary  to  stop 
the  robot  when  reporting  on  the  radio.  1 

Autonomous 

Quite  easy.  1 

Makes  it  much  easier  to  multi-task.  Reporting  over  the  radio  is  simplified 
on  account  of  my  having  both  hands  free.  1 

Overall  great  exercise.  1 

Much  easier  to  handle  secondary  tasks.  1 

It  was  easier  than  any  other  mode  because  I  can  just  react  to  secondary 
tasks  instead  of  anticipating  it.  1 

The  robot  navigating  on  it’s  on  was  more  efficient.  1 

Allowed  me  to  look  further  up  the  road  and  spot  lEDs  faster  while  also 
allowing  me  some  time  to  look  around  to  complete  the  secondary  tasks.  1 
Somewhat  frustrating,  but  good  practice  for  real  world  scenarios.  1 

Plotting  numbers  from  the  matrix  remain  the  most  difficult  because  it  takes 
the  most  time  and  attention  away  from  the  screen.  1 

My  eyes  were  taken  off  the  screen  for  at  least  3-4  seconds  when  reporting 
from  the  matrix.  It  seemed  impossible  for  me  not  to  miss  a  possible  IED.  1 
Reporting  the  numbers  was  still  time  consuming,  but  overall  much  better 
than  having  the  controls  to  worry  about.  1 
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Comments 


No.  of  Responses 


Only  problem  is  that  robot  will  choose  its  own  direction  to  travel.  If  you  miss 


an  item  of  interest,  you  cannot  turn  around  to  double  check.  1 

I  felt  like  I  was  more  focused  on  watching  the  path  of  the  robot  and  less  on 
the  secondary  tasks.  1 

Secondary  tasks  were  only  difficult  due  to  small  matrix  size  and  feeling 
pressured  to  look  quickly  while  the  robot  continued  to  move  forward,  so  no 
lEDs  would  be  missed.  1 

The  longer  I  had  to  search  for  numbers  or  the  person  of  interest  the  more  I 
felt  I  was  missing  potential  lEDs  on  the  course.  Quick  tasks  like  checking 
the  time  were  insignificant  breaks.  As  the  robot  drove  itself  I  could  afford 
to  only  look  for  lEDs  rather  than  focus  on  driving.  1 

The  camera  was  not  focusing  as  quickly  on  the  screen  as  needed  due  to 
light  from  sun.  1 

Harder  to  see  items  of  interest  on  the  course.  1 
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Intentionally  left  blank. 
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Appendix  D.  End  of  Experiment 
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SAMPLE  SIZE  =  27 


1 .  Please  indicate  your  1st,  2nd  and  3rd  choice  of  the  level  of  autonomy  to  use  for 
completing  the  following  tasks  by  placing  a  1 , 2,  or  3  in  the  appropriate  column. 

1  =  Best;  2  =  Second  best;  3  =  Worst 


Tasks 

Scale  above  -> 

Number  of  Responses* 

Teleoperation 

Semi- 

Autonomous 

Autonomous 

1 

2 

3 

1 

2 

3 

1 

2 

3 

Route  reconnaissance 

12 

7 

7 

5 

13 

9 

10 

6 

10 

Moving  around  obstacles 

14 

6 

6 

4 

13 

10 

9 

7 

10 

Assessing  the  route  for  navigability 

18 

1 

7 

3 

17 

7 

6 

8 

12 

Completing  recon  course  quickly 

11 

5 

10 

4 

11 

12 

12 

10 

4 

Performing  radio  tasks  while  using 
controller 

5 

11 

10 

2 

11 

14 

20 

4 

1 

Overall  route  reconnaissance 

13 

4 

9 

5 

10 

12 

9 

12 

5 

*  sample  size  varied  due  to  a  ‘no  response’  for  some  tasks 


Comments  No.  of  Responses 


Teleoperation 

Teleoperation  is  simple  and  efficient.  1 

It  is  easier  to  do  individual  task  in  teleoperation.  1 

Secondary  tasks  made  easier.  1 

Semi-autonomous 

Semi-autonomous  seemed  to  be  the  right  amount  of  help  from  robot  to 
confidently  navigate  course.  1 

Semi-autonomous  gave  robot  conflicting  instruction  from  user  and  sensors 
on  robot.  1 

Sensor  really  interfered  with  maneuverability.  1 

Autonomous 

Conducting  all  tasks  simultaneously  is  easier  in  the  autonomous  mode.  1 
Least  distracting,  but  made  more  specific  recon  difficult.  1 

General  comments 

Teleoperation  and  semi-autonomous  are  virtually  identical  in  most  respects 
to  radio  tasks  and  recon.  They  only  differ  in  level  of  control.  1 
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2.  Using  the  scale  below,  what  is  your  rating  of  the  characteristics  of  the  control  and 
display  system  that  you  used  during  this  experiment? 

1  2  3  4  5  6  7 

Extremely  bad  Very  Bad  Bad  Neutral  Good  Very  Good  Extremely  Good 


CHARACTERISTICS 

MEAN  RESPONSE 

Display  size  for  driving 

5.41 

Display  size  for  seeing  lEDs 

4.41 

Size  of  the  symbols  and  icons  on  the  screen 

5.56 

Display  layout 

5.70 

Map  fidelity 

5.26 

Appearance  of  the  graphic  user  interface  (GUI) 

5.63 

Size  of  the  buttons  on  the  controller 

6.33 

Overall  controller  size  for  use  in  the  field 

6.30 

Controller  configuration  for  use  in  the  field 

6.15 

Comments  No.  of  Responses 

Controller  is  perfect  size  for  operations  in  the  field.  1 

Very  easy  to  use.  3 

Have  had  experience  with  video  game  system  controllers.  1 

It  was  helpful  to  have  a  controller  that  is  familiar  to  people.  1 

Controller  should  be  smaller;  seems  too  bulky  for  use  in  the  field.  1 

Symbols  on  map  could  be  smaller.  1 

The  lEDs  were  difficult  to  identify.  Perhaps  it  was  due  to  the  graininess  of 
the  video;  sometimes  they  were  indistinguishable.  1 

Resolution  on  the  display  made  it  hard  to  spot  lEDs.  1 

Resolution  on  the  display  made  it  hard  to  spot  lEDs,  especially  in 
autonomous  mode.  1 

I  would  recommend  the  direction  cross  similar  to  the  Nintendo  8-bit  system 
controller  instead  of  the  mini  joystick.  1 

The  camera  resolution  was  very  poor  and  greatly  affected  by  ambient 
lighting.  1 

3.  What  are  the  ADVANTAGES  of  operating  the  robot  in  the  TELOPERATION  mode? 

Comments  No.  of  Responses 

Overall,  I  thought  this  was  the  best  mode.  1 

Self-control;  move  slow  or  fast,  not  just  one  speed.  1 

Control  and  predictability  of  movement  and  speed.  1 

Complete  user  control  increases  the  speed  at  which  the  course  can  be 
completed.  1 

Easier  to  maneuver.  1 

It  can  maneuver  quickly  around  obstacles.  1 
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Comments  No.  of  Responses 

Ease  of  movement  in  turns.  1 

Driving  is  easier.  1 

Fast.  2 

You  can  assess  situations/obstacles  that  a  computer  may  not  pick  up  on.  1 
Full  control  over  recon.  1 

Full  control  of  robot.  9 

No  need  to  worry  or  wonder  which  direction  it  will  turn.  1 

Easier  to  go  in  reverse  to  check  on  IED.  1 

Constant  choice  of  view.  1 

You  can  easily  investigate  objects.  1 

You  are  able  to  explore  the  course  more  and  search  for  objects.  1 

By  the  time  is  started  this  mode,  I  was  familiar  with  the  fell  of  the  controller, 
icons,  etc.  1 

Hands  free  available  for  secondary  tasks.  1 


4.  What  are  the  DISADVANTAGES  of  operating  the  robot  in  the  TELOPERATION 
mode? 


Comments  No.  of  Responses 

User  error  is  much  more  likely.  1 

Being  reliant  only  on  one’s  steering  ability.  1 

The  amount  of  brain  power  going  into  driving.  1 

Difficult  to  multi-task.  3 

You  must  stop  to  multi-task  in  proximity  to  obstacles.  1 

More  mistakes,  less  time  to  be  able  to  concentrate  on  all  objectives  of  the 
mission.  1 

I  may  let  my  guard  down  and  let  my  mind  wander.  1 

GPS  system  was  not  working  well.  This  skewed  the  map  fidelity 
downward.  1 

Slower.  2 

Less  confidence.  1 

I  may  not  feel  that  I  am  in  complete  control.  1 

Hitting  obstacles.  6 

In  an  obstacle  laden  environment,  teleoperation  mode  makes  it  easier  to  hit 
obstacles.  1 

You  will  hit  obstacles  if  you  don’t  pay  attention  at  all  times.  1 

No  way  to  avoid  obstacles  if  new  to  operating  system.  1 

Does  not  provide  corrective  obstacle/route  function.  I  ran  into  an  obstacle 
twice.  1 

Not  being  able  to  maneuver  around  all  the  obstacles.  2 

Navigating  around  single  obstacles  takes  longer.  1 

Very  easy  to  get  tunnel  vision  and  hit  objects.  1 

Collisions  more  likely.  1 

Takes  a  lot  of  focus.  1 


46 


Comments 


No.  of  Responses 


Harder  to  complete  secondary  tasks.  2 

Wasn’t  paying  attention  to  lEDs  as  much,  and  paying  attention  to  the  route.  1 
Must  be  more  skilled  with  the  handheld  controller.  1 

5.  What  are  the  ADVANTAGES  of  operating  the  robot  in  the  SEMI-AUTONOMOUS 

mode? 

None.  1 

Maintains  most  fine  control  advantages  of  teleoperation  that  autonomous 
lacks.  1 

Robot  self-corrects  at  obstacles.  3 

Total  control  and  freedom  to  look  around  if  needed  for  short  periods.  1 

It  will  not  allow  you  to  collide  with  anything.  2 

Confidence  in  not  hitting  obstacles.  4 

Makes  clearing/maneuvering  around  obstacles  easier.  8 

Moving  around  single  obstacles  is  more  fluid.  1 

Speed.  1 

You  can  move  faster  without  worrying  about  hitting  something.  1 

Can  just  hold  control  forward  while  conducting  a  secondary  task.  1 

Easier  to  perform  secondary  tasks.  1 

You  can  scan  more  easily  without  getting  tunnel  vision.  1 

You  can  multi-task.  1 

Easier  to  multi-task  without  having  to  worry  about  hitting  something.  1 

More  time  spent  looking  for  lEDs,  not  driving.  1 

If  given  practice  enough  to  become  accustomed  to  the  driving  technique 
required,  could  be  most  efficient.  1 

6.  What  are  the  DISADVANTAGES  of  operating  the  robot  in  the  SEMI-AUTONOMOUS 

mode? 

Comments  No.  of  Responses 

Complacency;  thinking  or  guessing  the  robot  will  stop  sometimes  when  it 
may  not.  1 

Not  having  full  control  of  the  robot.  2 

Harder  to  control.  1 

It  was  difficult  to  determine  when  I  was  controlling  the  robot  and  when  it 
was  controlling  itself.  1 

Have  to  navigate  around  objects  at  times.  1 

Robot  seemed  to  turn  away  from  obstacle  when  I  didn’t  want  it  to.  1 

Unpredictability  in  where  the  robot  heads  around  obstacles.  1 

Have  to  fight  the  controls  when  the  robot  is  wedged  in  between  obstacles. 

Robot  sometimes  has  a  preferred  route  when  it  finds  a  clear  route  ahead.  1 
You  are  unable  to  push  or  bump  obstacles.  1 

When  avoiding  obstacles,  it  sometimes  makes  it  harder  for  you  to  regain  2 
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Comments 


No.  of  Responses 


full  control  of  the  robot  to  place  it  back  on  track. 

Although  it  deviates  from  running  into  obstacles,  it  slowed  my  mission  time 
because  I  had  to  move  the  robot  back  on  route.  1 

Robot  tends  to  lag.  1 

Required  variable  levels  of  focus  depending  on  number  of  obstacles.  1 

Some  spaces  where  the  robot  can  fit  are  a  no-go  because  the  robot  thinks 
it  can’t  fit.  1 

Difficult  to  navigate  tight  spaces.  4 

Lack  of  freedom  of  movement  in  narrow  corridors.  1 

Less  time  to  concentrate  on  objectives  (i.e.,  quick  finish,  lEDs,  etc.).  1 

You  have  to  stop  when  your  attention  is  drawn  to  something  time 
consuming  to  investigate.  1 

Difficult  to  adjust  to  if  new  to  the  system.  1 

Jerky  at  times.  1 

More  difficult  to  maneuver.  1 

Angles  shift  too  far  on  their  own.  1 

Must  accommodate  speed  with  the  programs.  1 

Hard  to  tell  whose  turn  it  is.  1 

Takes  the  most  practice  to  become  accustomed  to  handling.  1 

Using  the  joystick  to  practically  wrestle  with  the  sensor  causes  frustration.  1 
Detracts  the  most  from  situational  awareness.  1 

7.  What  are  the  ADVANTAGES  of  operating  the  robot  in  the  AUTONOMOUS  mode? 

Comments  No.  of  Responses 

You  don’t  have  to  monitor  the  robot  the  whole  time.  1 

Easy  controls.  1 

Having  control.  1 

Full  control  over  how  the  robot  moves.  1 

Easy  to  navigate  objects  because  it  does  it  for  you.  1 

The  robot  completes  the  path  without  your  help.  1 

No  focus  need  be  spent  on  operation  of  the  robot.  All  side  tasks  can  be 
accomplished  while  the  robot  remains  in  motion.  1 

The  robot  handles  itself  and  avoids  obstacles  very  well,  allowing  the 
controller  to  have  the  most  situational  awareness.  2 

Fully  controlled;  ability  to  focus  on  one  part  of  the  mission  vs.  multi-tasking 
while  controlling  the  robot  and  telling  it  what  to  do.  1 

Took  the  stress  out  of  dealing  with  the  route  and  maneuver  the  robot.  1 

Ability  to  multi-task  very  easily.  8 

Takes  the  issue  of  operating  the  system  with  secondary  tasks  out  of  the 
equation.  1 

Allows  better  completion  of  secondary  tasks.  4 

I  did  not  have  to  operate  the  controller  and  was  able  to  focus  solely  on  my 
secondary  tasks.  1 
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Comments 


No.  of  Responses 


Conducting  secondary  tasks  seems  less  distracting.  1 

Spend  more  time  on  surveillance.  1 

Hands-free  operation.  1 

Speed.  2 

Easy  map  layout.  1 

Less  chance  of  hitting  an  object.  1 

Easier  to  spot  lEDs.  2 

A  better  look  at  suspicious  objects.  1 

Fewer  demands.  1 

Less  focus  is  required.  1 

8.  What  are  the  DISADVANTAGES  of  operating  the  robot  in  the  AUTONOMOUS 

mode? 

Comments  No.  of  Responses 

Using  the  computer  route  functions  can  be  a  disadvantage  if  not  properly 
trained  on  deleting  routes/points.  1 

None  here  at  training,  but  if  the  robot  makes  a  mistake  anything  could 
happen.  Subjective  question  considering  the  robot  made  no  mistakes.  1 

Very  easy  to  overlook  something  when  you  get  distracted,  since  the  robot 
doesn’t  stop  moving.  1 

Can’t  stop  to  look  around  as  quickly.  1 

You  stop  paying  close  attention  to  the  map,  which  could  be  a  potential 
issue.  1 

Might  not  go  the  way  you  need,  if  a  problem  occurs  rapidly.  1 

Lack  of  control.  4 

No  total  control  over  the  robot  which  makes  searching  for  individual  objects 
more  difficult.  1 

No  ability  to  slow  down  or  scan  the  area  at  will  to  better  see  the  ground.  1 

No  fine  controls  available  to  assist  in  identifying  lEDs  on  the  course.  1 

No  control  over  route  selection.  1 

You  do  not  have  control  over  the  robot  to  scan  for  lEDs.  1 

You  could  miss  an  IED  because  you  cannot  pause  to  get  a  decent  look  at 
something.  2 

Robot  continuously  moves  which  makes  identifying  lEDs  and  persons  of 
interest  more  difficult.  1 

Can’t  double  back  without  taking  control.  1 

Robot  moves  past  obstacles  without  you  seeing  them.  1 

Not  having  help  and  engaging  all  the  obstacles  by  yourself.  1 

Certain  situations  in  a  field  environment  may  require  humans  making 
decisions  vs.  a  computer  system.  1 

One  is  reliant  on  all  the  mechanisms  of  the  robot  to  function  properly.  1 

Tasks  are  more  difficult  to  report.  1 

I  may  have  a  harder  time  reacting  to  secondary  tasks.  1 
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Comments 


No.  of  Responses 


Poor  ability  to  perform  full  recon.  1 

Not  the  fastest.  1 

9.  What  needs  to  be  added  to  the  controller  to  give  you  more  knowledge  of  the  area 
around  the  robot? 

Overall  great  concept;  great  tool.  1 

Good  stuff;  it  seemed  to  suffice.  1 

A  mic  and  radio  in  case  in  the  semi-autonomous  or  teleoperation  mode.  1 
Wider  view  camera.  1 

Different  lens  so  less  fishbowl  effect  for  better  depth  perception.  1 

Lightening  of  shaded  imagery  and  higher  resolution,  with  fewer  video 
freezes,  which  are  disorienting.  1 

Zoom  feature  on  camera.  1 

More  cameras  (views).  1 

Better  resolution  for  the  camera.  2 

Camera  angle/direction  control.  1 

Ability  to  control  the  camera  to  move  in  every  direction.  3 

Ability  to  look  up  and  down.  1 

Swivel  the  camera  to  see  the  area  around  you.  1 

Ability  to  move  camera  by  itself  instead  of  the  whole  robot.  1 

Less  lag  time  on  camera.  1 

Ability  to  control  the  camera  to  see  if  you  are  going  to  hit  something.  1 

Ability  to  maneuver  the  camera  in  addition  to  the  robot  itself  would  be 
helpful.  1 

Something  similar  to  side  mirrors.  1 

More  sensitive  robot  directional  controls.  1 

Move  the  crosshairs  to  be  centered  as  one  drives,  not  centered  on  the 
camera.  1 

Maybe  before  initial  acceleration,  have  lines  pop  up  temporarily  to  show 
the  robot’s  borders.  1 

Maybe  a  rumble  pack  or  vibration  function  when  approaching  an  obstacles.  1 
An  indication  of  what  the  robot  senses  during  semi-autonomous  mode  so 
the  user  can  work  with  the  robot  to  resolve  collisions.  1 

More  peripheral  vision  should  be  available  to  the  controller  to  expand 
his/her  field  of  vision.  The  quality  of  the  video  would  also  be  helpful.  A 
crisper,  more  precise  video.  1 

10.  Overall  comments: 

Comments  No.  of  Responses 

Very  professional  training  and  instruction.  1 

Good  work!  1 

Good  piece  of  equipment.  1 
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Comments 


No.  of  Responses 


Enjoyable!  1 

It  was  an  experience  that  every  soldier  should  have  and  get  a  general  idea 
of  how  the  system  works.  1 

Great  robot.  Works  well  and  does  what  it  is  intended  to  do.  1 

Robot  is  easy  to  use  for  a  new  operator.  1 

Good  course.  2 

Good  overall  experiment  with  a  good  user  interface  and  joystick.  1 

Easier  to  spot  lEDs  with  some  measure  of  control.  Likely  due  to 
predictable  and  controllable  camera  jitter.  Similar  to  driving  a  car,  it  is 
easier  to  predict  shifts  and  motion  when  in  control  than  when  riding  as  a 
passenger.  1 

Suggest  upgrading  to  a  higher  definition  camera  to  see  picture  a  little 
clearer.  1 

The  robot  would  benefit  from  a  parallax  correctable  camera,  however,  in 
terms  of  it  making  it  easier  for  the  operator  to  account  for  the  size  of  the 
device.  1 
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Intentionally  left  blank. 
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List  of  Symbols,  Abbreviations,  and  Acronyms 


ANOVA 

analysis  of  variance 

ARL 

U.S.  Army  Research  Laboratory 

COTS 

commercial  off-the-shelf 

EMCCD 

Electron  Multiplying  Charge  Coupled  Device 

FLIR 

forward-looking  infrared  radar 

GPS 

global  positioning  system 

HRED 

Human  Research  and  Engineering  Directorate 

IED 

improvised  explosive  device 

ISR 

intelligence,  surveillance,  and  reconnaissance 

MOCU 

Multi-Robot  Operator  Control  Unit 

NASA-TLX  National  Aeronautics  and  Space  Administration-Task  Load  Index 
OA  obstacle  avoidance 

OCS  Officer  Candidate  School 

s  or  sec  second 

SA  situation  awareness 

SSC  Pacific  Space  and  Naval  Warfare  Systems  Center,  San  Diego  Pacific 
STORM  Small  Tactical  Optical  Rifle  Mounted 


UXO  unexploded  ordinance 
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NO.  OF 

COPIES  ORGANIZATION 


1  DEFENSE  TECHNICAL 
(PDF  INFORMATION  CTR 
only)  DTIC  OCA 

8725  JOHN  J  KINGMAN  RD 
STE  0944 

FORT  BEL  VOIR  VA  22060-6218 

1  DIRECTOR 

US  ARMY  RESEARCH  LAB 
IMAL  HRA 

MAIL  &  RECORDS  MGMT 
2800  POWDER  MILL  RD 
ADELPHI  MD  20783-1197 

1  DIRECTOR 

US  ARMY  RESEARCH  LAB 
RDRL  CIO  LL 
2800  POWDER  MILL  RD 
ADELPHI  MD  20783-1197 

1  DIRECTOR 

US  ARMY  RESEARCH  LAB 
RDRL  CIO  LT 
2800  POWDER  MILL  RD 
ADELPHI  MD  20783-1197 
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NO.  OF 

COPIES  ORGANIZATION 


NO.  OF 

COPIES  ORGANIZATION 


2  ARMY  RSCH  LABORATORY  -  HRED 
HUMAN  RSRCH  AND  ENGRNG 
DIRCTRT  MCOE  FIELD  ELEMENT 
RDRL  HRM  DW 
R  PETTITT 
C  CARSTENS 
BLDG  4  RM  332 

FORT  BENNING  GA  31905-5400 

1  ARMY  RSCH  LABORATORY  -  HRED 
RDRL  HRM  C  A  DAVISON 

320  MANSCEN  LOOP  STE  115 
FORT  LEONARD  WOOD  MO  65473 

2  ARMY  RSCH  LABORATORY  -  HRED 
RDRL  HRM  DI 

T  DAVIS 

J  HANSBERGER 

BLDG  5400  RM  C242 

REDSTONE  ARSENAL  AL  35898-7290 

1  ARMY  RSCH  LABORATORY  -  HRED 
RDRL  HRS  EA  DR  V  J  RICE 
2377  GREELEY  RD  STE  R 
FORT  SAM  HOUSTON  TX  78234-5002 

1  ARMY  RSCH  LABORATORY  -  HRED 
RDRL  HRM  DG  K  GUNN 
BLDG  333 

PICATINNY  ARSENAL  NJ  07806-5000 

1  ARMY  RSCH  LABORATORY  -  HRED 
ARMC  FIELD  ELEMENT 
RDRL  HRM  CH  C  BURNS 
BLDG  1467B  RM  336  THIRD  AVE 
FORT  KNOX  KY  40121 

1  ARMY  RSCH  LABORATORY  -  HRED 
AWC  FIELD  ELEMENT 
RDRL  HRM  DJ  D  DURBIN 
BLDG  4506  (DCD)  RM  107 
FORT  RUCKER  AL  36362-5000 

1  ARMY  RSCH  LABORATORY  -  HRED 
RDRL  HRM  CK  J  REINHART 
10125  KINGMAN  RD  BLDG  317 
FORT  BEL  VOIR  VA  22060-5828 


1  ARMY  RSCH  LABORATORY  -  HRED 
RDRL  HRM  AY  M  BARNES 
2520  HEALY  AVE 
STE  1172  BLDG  51005 
FORT  HUACHUCA  AZ  85613-7069 

1  ARMY  RSCH  LABORATORY  -  HRED 
RDRL  HRM  AP  D  UNGV ARSKY 
POPE  HALL  BLDG  4709 
BCBL  806  HARRISON  DR 
FORT  LEAVENWORTH  KS  66027-2302 

1  ARMY  RSCH  LABORATORY  -  HRED 
RDRL  HRM  DQ  MR  FLETCHER 
AMSRDNSCWSE  BLDG  3  RM341 
NATICK  MA  01760-5020 

1  ARMY  RSCH  LABORATORY  -  HRED 
RDRL  HRM  AT  J  CHEN 
12350  RESEARCH  PKWY 
ORLANDO  FL  32826-3276 

1  ARMY  RSCH  LABORATORY  -  HRED 
RDRL  HRM  AT  C  KORTENHAUS 
12350  RESEARCH  PKWY 
ORLANDO  FL  32826 

1  ARMY  RSCH  LABORATORY  -  HRED 
RDRL  HRM  AS  C  MANASCO 
SIGNAL  TOWERS 
BLDG  29808A  RM  303 
FORT  GORDON  GA  30905-5233 

1  ARMY  RSCH  LABORATORY  -  HRED 
RDRL  HRM  CU 
6501  E  11  MILERD  MS  284 
BLDG200A  1ST  FL  RMA1117 
WARREN  MI  48397-5000 

1  ARMY  RSCH  LABORATORY  -  HRED 
FIRES  CTR  OF  EXCELLENCE 
FIELD  ELEMENT 
RDRL  HRM  AF  C  HERNANDEZ 
3040  NW  AUSTIN  RD  RM  221 
FORT  SILL  OK  73503-9043 

1  ARMY  RSCH  LABORATORY  -  HRED 
RDRL  HRM  AV  S  MIDDLEBROOKS 
91012  STATION  AVE 
FORT  HOOD  TX  76544-5073 


55 


NO.  OF 

COPIES  ORGANIZATION 


1  ARMY  RSCH  LABORATORY  -  HRED 
RDRL  HRM  CN  R  SPENCER 
DCSFDI  HF 

HQUSASOC  BLDG  E2929 
FORT  BRAGG  NC  28310-5000 

2  ARMY  RSCH  LABORATORY  -  HRED 
HUMAN  RSRCH  AND  ENGRNG 
DIRCTRT  MCOE  FIELD  ELEMENT 
RDRL  HRM  DW  E  REDDEN 

6450  WAY  ST 

BLDG  2839  RM  310 

FORT  BENNING  GA  31905-5400 

1  ARMY  G1 
(CD  DAPE  MR  B  KNAPP 
only)  300  ARMY  PENTAGON  RM  2C489 
WASHINGTON  DC  20310-0300 


ABERDEEN  PROVING  GROUND 

5  DIRUSARL 
RDRL  HR 
L  ALLENDER 
T  LETOWSKI 
RDRL  HRM 

P  SAVAGE -KNEPSHIELD 
RDRL  HRM  A 
D  HARRAH 
RDRL  HRS  D 
B  AMREIN 
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